Systems and methods for learner growth tracking and assessments

ABSTRACT

Systems and methods for conducting automated skills mastery assessments in an e-learning environment may include assessing learner engagements with learning resources to produce mastery assessments, using historic interaction data derived through the engagements to train machine learning algorithm(s) to forecast evaluation outcomes of engagements based on engagement patterns indicative of skill fading, imparting learning, initial level of mastery, and/or a difficulty of acquiring mastery, applying the learning algorithm(s) to historic user interactions with learning resources to produce predicted evaluation outcomes, and, based on any differences between predicted outcomes and actual outcomes, refining parameter(s) of a mastery assessment parameter set used in calculating the mastery assessments, where a portion of the parameters correspond to attribute(s) of connections between the learning resources and skills of a skill hierarchy. The connections may be represented by logical indicators of relationships defined between the learning resources and the skills.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of and claims priority to U.S. patentapplication Ser. No. 17/716,944, filed on Apr. 8, 2022, which claimspriority to U.S. provisional patent application No. 63/172,433, filed onApr. 8, 2021, the entire contents of which are incorporated herein byreference in their entirety.

BACKGROUND

Digital learning systems present learning materials, such as text,video, audio, and/or interactive content, focused on teaching a learnerabout topics of interest. Some digital learning systems can dynamicallychange the presentation of content to the user based on the user'sindividual record of interacting with the digital learning system,thereby customizing content to the user's prior history with the digitallearning system. Often, learning content is presented to users by adigital learning system, and, based upon user interactions with thecontent, the digital learning system will score or grade the user'saccuracy or responses within the interactions. In illustration, adigital learning system may score a number of correct responses to anonline quiz. However, this focus on a particular piece of content limitsthe ability to trace mastery of skills. The inventors recognized theneed for an e-learning platform, systems, and methods providing theadvantages of a traceable path toward mastery of skills throughinteractions with digital learning content.

SUMMARY OF ILLUSTRATIVE EMBODIMENTS

In one aspect, the present disclosure describes systems and methods forenabling evaluation of a learner's mastery of skills based on thelearner's interactions with learning resources of an electronic learningplatform that are connected, via logical indicators of relationships(e.g., links, tags), with multiple skills per learning resource. In someembodiments, various systems and methods described herein enableevaluation of a learner's mastery of combinations of skills. Forexample, the systems and/or methods may enable consideration and/orexploitation of a hierarchical structure of skills such that mastery maybe evaluated on the skills that are higher in the hierarchy than (e.g.,ancestors of) the skills to which learning resources are connected.

The logical indicators of relationships between the learning resourcesand the skill hierarchy, in some embodiments, support amulti-dimensional learning model used to enhance development of multipleskills simultaneously, such as history and language learning ormathematics and science. Multi-dimensional learning models, for example,may improve skill mastery and learning retention through developing andstrengthening skills across learning disciplines. Assessments of skillmastery, in multi-dimensional learning models, may involve applyingfactors to the logical indicators of relationships that portion theimpact of certain learning resources among the skills developed orenhanced by that learning resource. For example, a strength factor maybe applied to a portion of the logical indicators of relationshipsrepresenting the impact of the linked skill relative to the whole of thee-learning resource. In illustration, an electronic learning resourcehaving content directed to both a history skill and a grammar skill mayinclude a first logical indicator of relationship (link or tag, as usedherein) connecting the electronic learning resource to the history skillwith a first strength factor and a second logical indicator ofrelationship connecting the electronic learning resource to the grammarskill with a second strength factor.

In some embodiments, mastery assessments are conducted to determineskill mastery in one or more skill areas. The mastery assessments may becalculated using a set of mastery assessment parameters, including atleast one parameter related to one or more attributes of the logicalindicators of relationships, such as the aforementioned strengthattribute.

In assessing mastery of skills in the e-learning platform, in someembodiments, a regression type machine learning algorithm may be appliedto data representative of historic learner engagements with the learningresources to derive patterns that, when applied to interactionsperformed by learners when engaging with the learning resources content,predict evaluation outcomes based on the interactions data. Thepredictions, in some examples, may be enhanced through machine-learningderived patterns related to skill fading, imparting learning, initiallevel of mastery, and/or difficulty of acquiring mastery.

Differences between actual outcomes and predicted outcomes, in someembodiments, are analyzed to determine adjusted mastery assessmentparameters. The impact related to the various mastery assessmentparameters (e.g., factors of the mastery assessment algorithm) can berefined, at this stage, to better align with the patterns derived fromthe historic data by the machine learning algorithm(s).

The foregoing general description of the illustrative implementationsand the following detailed description thereof are merely exemplaryaspects of the teachings of this disclosure and are not restrictive.

DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated in and constitute apart of the specification, illustrate one or more embodiments and,together with the description, explain these embodiments. Theaccompanying drawings have not necessarily been drawn to scale. Anyvalues or dimensions illustrated in the accompanying graphs and figuresare for illustration purposes only and may or may not represent actualor preferred values or dimensions. Where applicable, some or allfeatures may not be illustrated to assist in the description ofunderlying features. In the drawings:

FIG. 1A is a flow diagram of an example process for assessing skillsmastery in an electronic learning environment.

FIG. 1B is a block diagram of an example system architecture providingskills mastery assessment.

FIG. 2A is a flow chart of an example method for providing skillsmastery assessment.

FIG. 2B is a block diagram of an example partial hierarchy of skills andconnections between skills and learning resources.

FIG. 3A is a flow chart of an example method for scoring a skillsassessment.

FIG. 3B is a flow diagram of an example process for training scoringparameters for scoring a mastery assessment.

FIG. 4 is a flow chart of an example method for using knowledge tracingto propose new content to a learner based on historic masteryassessments.

FIGS. 5A and 5B are example graphical displays representing evidence ofmastery.

DETAILED DESCRIPTION

The description set forth below in connection with the appended drawingsis intended to be a description of various, illustrative embodiments ofthe disclosed subject matter. Specific features and functionalities aredescribed in connection with each illustrative embodiment; however, itwill be apparent to those skilled in the art that the disclosedembodiments may be practiced without each of those specific features andfunctionalities.

Reference throughout the specification to “one embodiment” or “anembodiment” means that a particular feature, structure, orcharacteristic described in connection with an embodiment is included inat least one embodiment of the subject matter disclosed. Thus, theappearance of the phrases “in one embodiment” or “in an embodiment” invarious places throughout the specification is not necessarily referringto the same embodiment. Further, the particular features, structures orcharacteristics may be combined in any suitable manner in one or moreembodiments. Further, it is intended that embodiments of the disclosedsubject matter cover modifications and variations thereof.

It must be noted that, as used in the specification and the appendedclaims, the singular forms “a,” “an,” and “the” include plural referentsunless the context expressly dictates otherwise. That is, unlessexpressly specified otherwise, as used herein the words “a,” “an,”“the,” and the like carry the meaning of “one or more.” Additionally, itis to be understood that terms such as “left,” “right,” “top,” “bottom,”“front,” “rear,” “side,” “height,” “length,” “width,” “upper,” “lower,”“interior,” “exterior,” “inner,” “outer,” and the like that may be usedherein merely describe points of reference and do not necessarily limitembodiments of the present disclosure to any particular orientation orconfiguration. Furthermore, terms such as “first,” “second,” “third,”etc., merely identify one of a number of portions, components, steps,operations, functions, and/or points of reference as disclosed herein,and likewise do not necessarily limit embodiments of the presentdisclosure to any particular configuration or orientation.

Furthermore, the terms “approximately,” “about,” “proximate,” “minorvariation,” and similar terms generally refer to ranges that include theidentified value within a margin of 20%, 10% or preferably 5% in certainembodiments, and any values therebetween.

All of the functionalities described in connection with one embodimentare intended to be applicable to the additional embodiments describedbelow except where expressly stated or where the feature or function isincompatible with the additional embodiments. For example, where a givenfeature or function is expressly described in connection with oneembodiment but not expressly mentioned in connection with an alternativeembodiment, it should be understood that the inventors intend that thatfeature or function may be deployed, utilized or implemented inconnection with the alternative embodiment unless the feature orfunction is incompatible with the alternative embodiment.

FIG. 1A illustrates a block diagram of an example system 100 forassessing skill mastery in an electronic learning (e-learning)environment. The system 100 involves an e-learning platform 102providing interactive learning resources 116 to one of a set of usersidentified by user profiles 114 through a platform graphical userinterface engine 108. Through user interactions 126 of an identifieduser 124 with elements within the learning resources 116, one or moreskills evaluation engines 110 and skills mastery assessment engines 112can assess and/or update progress in learning skills based on links,within the learning resources 116, to one or more individual learningresource elements 118, each corresponding to at least one skill within askills hierarchy 120. The assessments, for example, may be stored asskills assessments 122 linked to each individual user's profile 114.

The learning resources 116, in some implementations, are arranged in aweb site or web portal e-learning environment, as illustrated in anexample screen shot 130 at a user computing device 106 a. Individuallearning resources, in some examples, can include one or more quizzes,videos, readings, simulations, data manipulatives, online drawingactivities, graphical coding modules, interactive learning modules,information or notes arranging modules, and/or games. As illustrated,the example screen shot 130 includes a set of learning resources 134 forselection, including a link to information 134 a (e.g., one or moreadditional and/or external resources for learning about the subject“pronouns and be”), a writing activity 134 b (e.g., “write it?”), a datamanipulative 134 c (e.g., “words, words, words!”), a reading 134 d(e.g., “read it!”), a game 134 e (e.g., “play it!”), and an interactivelearning module 134 (e.g., “hear it, say it!”). The learning resources134, for example, may be part of a foreign language instruction, Englishas a second language instruction, or early learning instruction.

As the user at the computing device 106 a interacts with one of thelearning resources 134, in some implementations, the e-learning platform102 gathers interactions 126 with learning resource elements 118 withinthe selected learning resource 134 and associates the interactions 126with a user identification 124 of the user logged into the e-learningplatform 102 via the computing device 126. The learning resourceelements 118, for example, can include individual questions, data entryfields, game levels or skills completions within an ongoing game typelearning resource 116. In other words, interactions with learningresource elements 118 can include, in some examples, typed responses,utterances (e.g., in the verbal interactive learning module 134),gestures (e.g., captured by a video camera of the computing device 106 aor an accelerometer of an input device connected to the computing device106 a), selections of controls within the user interface 130, movementof interactive elements within the user interface 130, or submission ofactivities (e.g., drawings, maps, code segments, etc.) through thelearning resources 116. Further, interactions data can include timing ofinteractions, such as how long the learner took to answer a questionafter presentation of the question or a length of time the learner spentsolving a problem, such as performing iterative moves in an interactivechallenge. In an additional example, interactions data can include aportion presented to the learner, for example a portion of a readingdocument scrolled through by the learner while reading or a portion ofan educational video played.

In some implementations, one or more skills evaluation engines 110 matcheach interaction to an individual element with at least one skilllogically connected to the learning resource element in a skillshierarchy 120 via logical indicators of relationships. The skillsevaluation engine(s) 110, for example, can include engines designed toevaluate skills based upon different styles of interactions, such as theexample interactions described above. The skills evaluation engine(s)110 may log the results of the assessment of the user interactions 126as a skill assessment 122 of the user 124 as linked to one of the userprofiles 114.

In building skills via the e-learning platform 102, in someimplementations, one or more skills mastery assessment engine(s) 112 mayanalyze skills assessments 122 individually and/or over time to derivean ongoing mastery assessment 128 related to the user having useridentification 124. The mastery assessment 128, for example, may beprovided to a computing device 106 b (e.g., the student computing device106 a, a teacher computing device, and/or a parent or other supervisinguser computing device) for presentation as a mastery assessmentgraphical user interface 132. The mastery assessment 128, in someexamples, may include relevant times (e.g., a timespan, one or moretimestamps, etc.) of interaction with the e-learning platform 102 towork on skills of a particular type or subject (e.g., sub-hierarchy), alast learning resource element of interaction, a current mastery value,an evaluation of mastery over time (e.g., bar graph, line graph, etc.)and/or mastery confidence interval information. An example masteryassessment 128 is presented, for example, in FIG. 5A, described ingreater detail below.

FIG. 1B is a block diagram of an example system architecture andenvironment 150 for providing skills mastery assessment. The system 150,in some implementations, includes an electronic resource data repository158, a user data repository 160, and a collection of engines of thee-learning platform 102. Students 152 (e.g., learners), instructors 154,and/or learning supervisors 156 of the students 152 (e.g., parents,guardians, other family members, daycare center personnel, nannies,etc.) may access the e-learning platform 102 to interact with thecapabilities provided by the assortment of engines. For example, thestudents 152, instructors 154, and/or learning supervisors 156 may logon through a learning resource GUI engine 108 a and/or an assessment GUIengine 108 b, using information stored to a relevant profile, such asthe student profiles 182, instructor profiles 184, and supervisorprofiles 192 to access capabilities of the e-learning platform. Forexample, learning supervisors 156 may only have access to the assessmentGUI engine 108 b, while instructors 154 may have access to review workand/or provide online comments via the learning resource GUI engine 108a as well as accessing the assessment GUI engine 108 b to track studentprogress. Some students may only have access to the learning resourceGUI engine 108 a, such that the students are not privy to the masteryassessments provided by the e-learning platform 102. Other students mayhave access to both the learning resource GUI engine 108 a and theassessment GUI engine 108 b.

In some implementations, students interact with learning resources 170,each including one or more learning resource elements 172, via thelearning resource GUI engine 108 a, as described in relation to FIG. 1A.

In some implementations, during and/or upon completion of interactingwith a particular learning resource 170, a skills evaluation engine 110receives student interactions 190 with learning resources 170 andassesses progress of the student 152 in one or more skills areas, asdiscussed in relation to FIG. 1A. The skills areas, for example, may beidentified from within a skill hierarchy 174 logically linked to thelearning resource elements 172 of the learning resources 170. The skillsevaluation engine 110, for example, may be used to present to thestudent 152 via the learning resource GUI engine 108 a a scoring orgrading of the student's interactions with the learning resources. Thescoring or grading, for example, may be stored as a skill evaluation186.

In some implementations, a skills mastery assessment engine 112 obtainsthe scorings or gradings from the skills evaluation engine 110 andcalculates mastery of skills. Further, if the student has prior masteryassessments 188 and/or skill evaluations 186 stored that are related tothe same skill(s), in some embodiments, the skills mastery assessmentengine 112 calculates the mastery assessment according to masteryassessment parameters 180 corresponding to features of the skillevaluations 186 and/or metrics derived therefrom. The features, in someexamples, can correspond to aspects indicative of one or more of skillfading, imparting learning, initial level of mastery, and/or difficultyof acquiring mastery, as discussed in further detail below.

In some embodiments, the skills mastery assessment engine 112 calculatesthe mastery assessment based in part on skill-item weights 176 (e.g., aweight of the connection between a given resource element 172 and agiven skill tagged or linked to the resource element 172 via a logicalindicator of relationship) and/or skill-item strengths 178 (e.g., astrength of the connection between a given resource element 172 and agiven skill tagged or linked to the resource element 172). Theskill-element strength 178, for example, may represent a relevance ofthe skill to the individual resource element 172 (e.g., as opposed toother skills linked to the same resource element 172). The skill-itemweight 176, for example, may represent an amount of learning impactprovided by the content (e.g., how deeply or intensely focused a givenresource element 172 is on imparting and/or assessing knowledge relatedto the given skill). The skills mastery assessment engine 112, forexample, may access mastery assessment parameters 180 to identifyalgorithms and/or equations for applying factors to calculating themastery assessments 188 including how to apply the skill-item weights176 and/or the skill-item strengths 178.

Historic interaction data (e.g., derived from the student interactionsdata 190), in some embodiments, is provided to an evaluation predictionengine 162 for predicting mastery assessments based on historic data andcurrent mastery assessment parameters 180. The evaluation predictionengine 162, for example, applies statistical analysis and/or machinelearning to adjust the mastery assessment parameters 180 to best matchdemonstrated skill mastery derived from historic skill evaluations 186,mastery assessments 188, and/or student interactions 190. The evaluationprediction engine 162, for example, may produce one or more adjustedparameters to the mastery assessment parameters 180.

A content recommendation engine 164, in some embodiments, analyzesmastery assessments 188 to determine future content to recommend to thelearner. The content recommendation engine 168, for example, mayidentify learning resources 170 and/or learning resource elements 172tagged or linked to skills within the skill hierarchy 174 that wouldprovide the learner with strengthening of developing skill sets.

FIG. 2A is a flow chart of an example method 200 for organizing a dataarchitecture supporting skills mastery assessment of learning elementswithin an e-learning platform. The method may be performed manually,semi-automatically, or automatically, based in part on a foundation ofthe data architecture in a given system. The method 200, for example,may be performed in part by the e-learning platform 102 of FIG. 1A andFIG. 1B.

In some implementations, the method 200 begins with obtaining learningresources (202). As discussed above, the learning resources, in oneexample, can include one or more assessment questions. In illustration,assessment questions can include an inquiry, such as “What is thederivative of sin(x)?” Further, assessment questions can include wordproblems, such as “A car is moving at a constant speed of 20 m/s along acurving road. The curvature radius of the road is 130 m. Find theacceleration of the car.” The learning resources can include videos. Forexample, a learning resource can be a video of a teacher presenting atopic or a video of a person performing a science experiment. Thelearning resources can include one or more readings. For example, thereadings can be one or more excerpts from a textbook. The learningresources can include simulations, such as a simulation of solidchanging to a gas The learning resources can include one or more datamanipulatives. In illustration, a manipulative can provide aninteractive online exercise where the user adjusts the positioning of atennis racket, the tension of its strings and the direction of theracket swing in order for the tennis ball to hit the target at aprescribed speed and with a prescribed spin. The learning resources caninclude games. For example, of the games can include a math game wherethe user receives points for correct answers and advances throughchallenges involving characters and potentially a plot or story line.

In some implementations, if a given learning resource includes multiplelearning elements (204), the learning resource is separated intoindividual learning elements (206). For example, a quiz can containmultiple questions, where each question is a separate learning elementavailable for later assessment. In a further example, a learning gamemay be separated into game levels or experience types within a game(e.g., whole numbers vs. fractions in a math game). As illustrated in anexample learning resource structure 254 of FIG. 2B, for example, thelearning resource 254 can be separated into two elements 254 a, 254 b.

In some implementations, the learning resources/resource elements areeach categorized into one or more groups according to a mastery effectderived through interaction with the learning resource. If, for example,their nature is different enough so that the mastery effect frominteracting with them is expected to be substantially different.Examples of groups can include, in some examples, content-typegroupings, such as an instructional videos group and a questions group.In another example, groups can include groups by difficulty level, wheremore difficult learning resource elements may be treated differently indetermining the mastery assessment.

In some implementations, a set of skills having a hierarchical structureare obtained (210). Skill within the hierarchical structure can beassigned a “parent” skill and/or one or more child skills, thus encodingthe hierarchical structure among skills. In some embodiments, each skillhas no more than one parent skill, but the same parent skill may beassigned to any number of other skills. A skill can be referred to as a“child” skill in relation to its parent skill. More generally, skills inthe hierarchy can be referred to as “ancestors” or “descendants” of eachother (e.g., the parent of a parent is an ancestor, and children of achild are descendants). An example partial skills hierarchy 252 isillustrated in FIG. 2B, including skills 256-272. Skill 6 266 isillustrated as having two child skills, skill 6.A 274 a and skill 6.B274 b (e.g., sibling skills).

In some embodiments, the hierarchical skill structure is created by orbased on a teaching standard. For example, the following two skills areparent and child skill levels derived from the Next Generations ScienceStandards (NGSS), which provides a hierarchical structure:

-   -   LS1.B: Growth and development of organisms.    -   LS1.B.2: Plants reproduce in a variety of ways, sometimes        depending on animal behavior and specialized features for        reproduction.

In some implementations, each learning resource or element thereof islogically connected to one or more skills of the hierarchical skillstructure (212). The connections, for example, may be referred togenerally as logical indicators of relationships. Connecting theindividual learning resources/elements to the one or more skills, forexample, can involve linking, within a database or other data network,each learning resource/element record to one or more skill records. Inillustration, an individual science question element can be taggedwithin an American learning standards skill structure, an internationallearning standards skill structure, a Canadian learning standards skillstructure, etc. In this manner, the same learning elements may beapplied to determining mastery based upon one of a number of skillsmastery formulations. For example, as illustrated in FIG. 2B, learningresource element 1 254 a has been tagged with Skill 6.A 274 a. Further,learning resource element 2 254 b has been tagged with Skill 6.A 274 aand Skill 6.B 274 b.

In another example, in a multi-dimensional (cross-discipline) learningstandard structure, such as the NGSS, a same learning resource elementmay be tagged for two or more disciplines applied to learning theparticular skill. In illustration, a multi-dimensional learningstandard-supporting learning resource element presenting content fordeveloping skills related to climate, including a mathematical skilltag, a weather science skill tag, and a literacy skill tag. Inillustration, as shown in FIG. 2B, a learning resource element 3 286 bis linked or tagged to skill 1 256 with a weight 4 288 as well as toskill 3 260 with a weight 5 290.

In some implementations, a strength is applied to at least a portion ofthe tags (214). For example, the strength may indicate a strength ofpresentation of the skill within the tagged item (learningresource/element). In illustration, a video focused on a particularskill may receive a strong indication of strength, while another videothat weaves the skill into supporting the presentation of a differentskill may receive a weaker indication of strength. The strength, forexample, can be a numeric value between 0 and 1, between 1 and 10, orbetween 1 and 100. As illustrated in FIG. 2B, for example, a strength S1292 corresponds to a tag between learning resource element 3 286 andskill 1 256, while a strength S2 294 corresponds to a tag betweenlearning resource element 3 286 and skill 3 260.

In some embodiments, a strength between an item (learning resource orlearning resource element) and a skill may be designated by a number ofconnections between the item and the skill. For example, a neuralnetwork or other linked data network may express strength in the form ofnumber of linkages. As illustrated in FIG. 2B, learning resource element1 254 a is logically connected to skill 6.A 274 a through a single link276. Conversely, learning resource element 254 b is logically connectedto skill 6.A 274 a through a set of three logical links 278 a-c.

In some implementations, a weight is determined for at least a portionof the tags (216). The weight, for example, may be a numeric valueindicating a relative strength of evidence of mastery that aninteraction with the respective element or learning resource (e.g.,item) carries. For example, a depth of knowledge (DOK) is a commoncharacteristic of learning resources that can be applied as a weightvalue. As illustrated in FIG. 2B, each link between learning resourceelements 254 a and 254 b includes an associated weight 280, 282 a-c,284.

Although illustrated as a particular series of operations, in otherimplementations, the method 200 may include more or fewer operations.For example, strengths may not be determined for tags (214) and/orweights may not be determined for each tag (216). Further, in someimplementations, one or more operations may be performed in a differentorder and/or in parallel. For example, weights may be determined (216)prior to applying strengths to each tag (214). In another example, thelearning resources may be grouped after applying strengths (214) and/orweights (216). Other modifications of the method 200 are possible whileremaining within the scope and spirit of the disclosure.

Turning to FIG. 3A, a flow chart presents an example method 300 fordetermining evidence of mastery of skills within a skill hierarchy usinga linked learning resource and skill hierarchy data architectureestablished, for example, through the method 200 of FIG. 2A. Portions ofthe method 300, for example, may be performed by the skills evaluationengine 110 and/or the skills mastery assessment engine 112 of FIG. 1B.

In some implementations, the method 300 begins with obtaining inputscorresponding to a learner's interactions with one or more learningresources (302). The inputs, in some examples, can include answers toquestions, completion of tasks, and/or scores/levels achieved in playinga game. For example, the inputs may relate to each user submissionrelevant to individual evaluation, such as clicking a selection of amultiple-choice answer or entering a series of adjustments to aninteractive model. The inputs, for example, may be obtained by thelearning resource GUI engine 108 a of FIG. 1B.

In some implementations, the inputs are evaluated in accordance withevaluation rules related to desired/undesired interactions and/orcorrect/incorrect responses to the items (learning resources and/orelements thereof) (304). Some inputs, for example, may includegraded/scored interactions, such as answers to quiz questions. Someinputs, in another example, may include ungraded/unscored interactions(e.g., completed or not completed), such as playing a video to itsentirety or dwelling on a reading element long enough to have morelikely than not read the content. As such, at least some of the inputsare evaluated based on evaluation rules pertaining to partial credit forachieving a portion of the goal(s) of the learning element. In otherexamples, partial credit may be associated with completing part of alearning game (e.g., running out of attempts prior to completion of agame level) or completing a portion of a quiz. Evaluating the inputs,for example, can include “grading” activities performed within thee-learning platform. The evaluating may be performed, for example, bythe skills evaluation engine 110 of FIG. 1B.

In some implementations, the interactions and/or responses (e.g., the“graded content”) are correlated with corresponding skills (306). Forexample, the links or tags between the items (learning resources and/ortheir individual elements) and skills in a skills hierarchy areidentified to assess progress in relation to skills of the skillshierarchy. For each of the one or more skills that a particular item(learning resource or element thereof) is tagged with, a change in themastery level of the respective skill can be determined. The skillsmastery assessment engine 112 of FIG. 1B, for example, may correlate theinteractions and/or responses with the corresponding skills.

In some implementations, if the tags or links between each item and oneor more corresponding skills have an applied strength (e.g., asdescribed in relation to operation 214 of the method 200 of FIG. 2A)and/or weight (e.g., as described in relation to operation 216 of themethod 200) (308), the weight and/or strength is applied to theevaluated inputs to calculate adjusted interactions/responses (310). Forexample, numeric values representing an evaluation of each input may beweighted, multiplied, or otherwise altered by a factor representative ofthe strength and/or weight of the tag. Turning to FIG. 2B, since thestrength between learning resource element 2 254 b and skill 6.A 274 ais represented as three separate links, applying this strength mayinvolve multiplying the evaluation of the input by three (e.g., thisanswer is worth three times the value of other answers in demonstratingmastery of the skill 6.A 274 a). Further, a weight W2 282 a-c may beapplied to one or all of the links representing strength, such that theweight adjusts the evaluation value by a relative importance of skill6.A to the learning resource element 2 254 b.

In some embodiments, application of weights and/or strengths may differbased on whether the learner entered a correct answer or an incorrectanswer. For example, only correct answers may be magnified by thestrength factor, while all answers are magnified by the weight factor.Other combinations are possible.

In some implementations, for each skill, the corresponding evaluated oneor more inputs is used to calculate a mastery level for the learner inthe skill (312). The mastery level, for example, represents a relativegrasp of the subject matter of a given skill within the skill hierarchy.The mastery level may be calculated, in some examples, by determining amedian, mean, or weighted average of the values of the evaluated inputsin each respective skill. Mastery level, in further examples, may becalculated in part on one or more factors (e.g., a portion of themastery assessment parameters 180). The factors, in some examples, caninclude a difficulty of the learning resource element (e.g., amplifyingpositive scores/values for difficult learning elements) and/or one ormore medical factors of a learner's student profile 182 (e.g., alearning disability, neurological disorder, physical hindrance, or otherimpediment that may modify the learner's patterns of interactions inrelation to other learners within a same group (e.g., age, grade, etc.).

In some implementations, for each skill, a confidence value representinga confidence in the learner's present level of mastery of the skill iscalculated (314). The confidence value, in some examples, may include aconfidence interval (e.g., +/−a margin surrounding the calculatedmastery assessment), a percentage confidence, or a confidence rating(e.g., high confidence, medium confidence, low confidence, etc.).

Although illustrated as a particular series of operations, in someembodiments, the operations of the method 300 are performed in adifferent order and/or one or more of the operations of the method 300are performed in parallel. For example, the operations 310, 312, and 314may be performed in parallel for each skill of the one or more skills.In some embodiments, the method 300 includes more or fewer operations.For example, the method 300 may include calculating mastery assessmentmetrics based on change in mastery level over time, such as a rate ofincrease in mastery level. Other modifications of the method 300 arepossible while remaining in the scope and spirit of the disclosure.

In some implementations, the mastery level is presented for review by auser, such as the learner, a supervisor, or a teacher. If past masteryassessments 188 are available, mastery assessments may be presented overtime to demonstrate the learner's progress in mastering the subjectskill.

In one example, turning to FIG. 5A, a screen shot of an example masteryassessment graph 500 presents a synopsis of a learner's progress inmastering a particular skill as visualized in relation to achievement502 (e.g., percentage accuracy of user inputs/percentage masterydemonstrated) over time 504. Each of the illustrated score indicators(circles) 506 represents a score received (e.g., scaled from 0 to 1),with a size of each score indicator 506 representing the relativerelevance of the score to the skill (e.g., a tagged strength of thelearning resource element corresponding to the score and/or a taggedweight of the learning resource element corresponding to the score). Amastery level plot 508 is represented by a solid line upon the graph500, illustrated movements within the learner's achieved mastery levelover time. As illustrated, the learner began interacting with theplatform with a relatively low mastery level (e.g., appearing to be alittle less than 0.1) and over time (e.g., from a day/time 0 to aday/time 100) has achieved a mastery level near 1 (e.g., near completemastery). A band 510 surrounding the mastery level plot 508 represents aconfidence interval for the mastery level at each point along themastery level plot 508. In some embodiments, the time represents a totaltime the learner has spent working on the particular skill within thee-learning platform, such as 100 days, 100 hours or 100 minutes.

In some implementations, the mastery level determination is trained foroptimal performance, thereby maximizing its predictive power, usinghistorical data collected by the e-learning platform to refine masteryassessment parameters. Turning to FIG. 3B, a flow diagram illustrates anexample process 320 for training the mastery assessment parameters 180using historic interaction data 190. In some embodiments, the masteryassessment parameters 180 include factors such as student age, studentregion, student grade level, and/or learning standard that factor intoone or more algorithms for calculating, from skill evaluations, themastery assessment. The mastery assessment parameters, 180, in someembodiments, includes multiple sets of mastery assessment parameters.The mastery assessment parameters 180 may be divided into parameters pergroup of a set of learner groups. Learners may be separated into groups,in some examples, based on particular learning standards (e.g., a U.S.public school skills hierarchy, a Canadian public school skillshierarchy, a per school district standard, etc.), grade level, age orage range (e.g., early grade school range, later grade school range,middle school range, high school range, etc.), and/or geographic region(e.g., city, county, zip code, area code, state, province, etc.).

In some implementations, historic interaction data 322, including, insome examples, skills evaluations 186, user interactions 190, and/ortimestamps corresponding to at least a portion of the user interactions190 (e.g., beginning and ending/time elapsed for taking quiz, time ofsubmission of an answer to a question, etc.) is supplied to one or moremachine learning algorithms 326 of the evaluation prediction engine 162.The user interactions 190, in a further example, can include a number ofactions taken during the interaction (e.g., how many individualadjustments applied to an interactive physics model to achieve asuccessful result, a number of “lives” used in achieving conclusion of alearning game, a number of times a video presentation was paused, etc.).The historic interaction data 322, in another example, may includehistoric mastery assessment metrics 328 such as, in some examples, aninitial mastery level related to each skill, a rate of improvement inmastery level related to each skill, a length of time to progressbetween mastery levels, and/or a length of time elapsed between masteryassessments. The historic interaction data 322 may be correlated toskills within the skills hierarchy 174 and/or to tagged learningresources/learning resource elements 324 (e.g., to access skill-elementweights 176 and/or skill-element strengths 178 as discussed in relationto FIG. 1B).

In some implementations, training data 322 provided to the machinelearning algorithm(s) 326 includes interaction data grouped by skillfamily 322 a (e.g., including ancestors and descendants along a branchof the skills hierarchy 174), interaction data grouped by skill 322 b,and/or interaction data grouped by content type 322 c (e.g., videos,quizzes, games, etc.). For example, different machine learningalgorithms 326 may be better suited to assess different types ofhistoric interaction data 322 (e.g., math & science intensive incomparison to foreign language learning, games in comparison to quizzes,etc.).

The machine learning algorithm(s) 326 are designed to forecastevaluation outcomes based on student interaction data 190. The machinelearning algorithm(s) 326, in some embodiments, include a regressionmodel, such as a K-nearest neighbor (KNN) regression model or a supportvector regression (SVR) model. The regression model, in other examples,may apply a moving linear regression, a weighted moving average, or anexponentially weighted moving linear regression.

In some embodiments, forecasting evaluation outcomes using the machinelearning algorithm(s) 326 includes predicting, based on patterns ofstudent behavior, future movement in achievement in interacting withlearning resource elements 172. These patterns of student behavior, insome examples, can take into account the impact on student achievementby factors such as fading, imparting learning, initial level of mastery,and/or difficulty of acquiring mastery. For example, the machinelearning algorithm(s) 326 may detect patterns related to a time it takeslearners to progress through levels of mastery (e.g., a rate of learningeffect on progressing to mastery of a skill) In another example, themachine learning algorithm(s) 326 may detect patterns related to theimpact of initial familiarity with a skill on a learner's futureprogress through levels of mastery. Initial skill familiarity, in someexamples, may be found by the machine learning algorithm(s) 326 toinvolve the first score corresponding to a particular skill of the skillfamily (ancestors and descendants), the first score corresponding to oneof the learning resource elements 172 linked as having strong relevanceto the particular skill (e.g., based on skill weightings 176 and/orskill strengths 178), or a first N (e.g., 2, 3, up to 5, etc.) scorescorresponding to the particular skill In a further example, the machinelearning algorithm(s) 326 may detect patterns related to an amount oftime it takes a learner to progress from the original interaction tomastery of the particular skill (e.g., a rate of learning effect onprogressing to mastery of a skill) which may be indicative of adifficulty in learning the skill. Further in relation to a learner'sdifficulty in acquiring mastery of a particular skill, the machinelearning algorithm(s) 326 may detect patterns related to a number ofrepetitions with learning resource elements. In another example, themachine learning algorithm(s) 326 may detect patterns related to theeffect fading has on skill acquisition due to failing to exercise theskills on a regular basis over time. The training data 322, in anillustrative example, may include timestamps useful in derivingfrequency of engagements with learning resources 170/learning resourceelements 172 associated with a particular skill family of the skillhierarchy 174.

The predicted evaluation algorithm(s) 330, in some implementations,intake historic student interactions data 190 and apply the machinelearning algorithms 326 to predict skill evaluations, resulting inpredicted outcomes 332.

In some implementations, the predicted outcomes 332 are applied by amastery assessment parameter refinement module 334 to refine currentmastery assessment parameters 180 to better align with learners' actualachievement data as analyzed by the machine learning algorithms 326. Byapplying the historic interaction data 322 to predict the outcomesachieved by a user on any subset of learning resource elements andcomparing the predicted outcomes 322 with the skill evaluations 186received by learners in reality, the mastery assessment parameterrefinement module 334 of the evaluation prediction engine 162 mayre-tune the mastery assessment parameters 180 to maximize the predictionaccuracy of the mastery assessments 188. The mastery assessmentparameter refinement module 334 may output one or more adjusted masteryassessment parameters 336 for replacing one or more of the parameters ofthe mastery assessment parameters 180. The adjustments in the masteryassessment parameters 180, in some examples, can include an adjustedweight for applying a particular parameter to the mastery assessmentcalculation algorithm, an adjusted sub-algorithm (e.g., logarithmicapplication of the particular assessment parameter 180 rather thanlinear application of the particular assessment parameter 180) forapplying a particular parameter to the mastery assessment algorithm,and/or an adjusted rule for applying a particular parameter to themastery assessment algorithm (e.g., in cases where a particular timingfactor is greater than N days, etc.).

In some embodiments, the mastery assessment parameter refinement module334 may propose an additional parameter to include within thecalculations of the skills mastery assessment engine 112 of FIG. 1B torefine analysis of the skill evaluations 186. In an initial example, iffactoring in certain timing data elements of the student interactionsdata 190 would lead to better accuracy of the mastery assessments 188,the mastery assessment parameter refinement module 334 may proposeinclusion of such timing data in the mastery assessment parameters 180.

The process 320, in some embodiments, is performed periodically. Forexample, the process 320 may be performed monthly, bi-weekly, weekly, ordaily. In some embodiments, the process 320 is performed for each set ofmastery assessment parameters. Individual sets of mastery assessmentparameters, for example, can correspond to individual learner groupsassessed by the electronic learning platform (e.g., age, grade level,and/or region, etc.) and/or individual skill hierarchies used forevaluation purposes within the electronic learning platform (e.g., basedon different learning standards) to customize assessments for differentlearner experiences and/or different instructor or learning supervisordesires. A same learner may be assessed under multiple learningstandards and/or as part of multiple learner groups, providing theability for the e-learning platform to present apples-to-applescomparisons of engagement outcomes across swaths of learners engagingwith the e-learning platform. The training data 322, in someimplementations, is filtered based on one or more criteria, such as athreshold length of time a corresponding learner has been engaging withthe e-learning platform and/or a threshold mastery assessment levelacquired by the e-learner in a subject skill or skill family. A sameunderlying type of machine learning algorithm 326, therefore, can betrained using multiple variations of sets of training data to deriverefined mastery assessment parameters focused to a select subset of thelearners of the e-learning platform.

FIG. 4 is a flow chart of an example method 400 for using knowledgetracing to propose new content to a learner based on historic masteryassessments. At least a portion of the method 400, for example, may beperformed by the content recommendation engine 164 of the e-learningplatform 102 of FIG. 1B. The method 400, for example, may result ininformation being presented to a learner via the learning resourcegraphical user interface engine 108 a of FIG. 1B.

In some implementations, the method 400 begins with obtaining skillevaluations and/or mastery assessments corresponding to one or moreskills (402). For example, historic mastery assessments 188 and/or skillevaluations 186 for a given student profile 182 related to a skillfamily of the skill hierarchy 174 (e.g., descendants of a particularskill branch, as described in relation to FIG. 2B) may be accessed fromthe user data store 160 of FIG. 1B.

In some implementations, the skill evaluations and/or the masteryassessments are reviewed for the target skill family to identify one ormore weaknesses the learner exhibits within the skill family hierarchy(404). The weaknesses, for example, may involve a particular skill, suchas skill 6.A 274 a of the skills hierarchy 252 of FIG. 2B, or a branch(e.g., skill 6.A 274 a and any descendants thereof). The weaknesses, inone example, can include missing skill evaluations (e.g., the learnerhas not yet been presented with any learning resources 170/resourceelements 172 related to a particular skill or skills of the skillfamily). In another example, the weaknesses can include low skillsevaluations related to learning resources 170/resource elements 172involving one or more particular skills of the skill family (e.g.,evaluations having a value beneath a threshold criteria, evaluationshaving a value below a typical evaluation value for the particularlearner, etc.). In a further example, the weaknesses can include lowskills evaluations related to particular types of learning resources170/resource elements 172 involving one or more particular skills of theskill family (e.g., the learner appears to struggle with word problemsrelated to the math skill family).

In some implementations, if weaknesses are discovered within the skillfamily (406), learning content is identified within the skill familybased on the identified weakness(es) (408). The learning content, forexample, can include learning resources 170/resource elements 172 taggedwith the skill identified as needing strengthening. The learning contentidentified may include content having at least X threshold weight (e.g.,according to skill-element weights 176 of FIG. 1B) and/or Y thresholdstrength (e.g., according to skill-element strengths 178 of FIG. 1B) toensure that the content is valuable in bolstering the learner'scomprehension of the skill. In some embodiments, if the learner hasreviewed all relevant content related to the skill, the learning contentmay be content the learner has not engaged with in a while and/orcontent the learner was less successful with during engagement (e.g.,skill evaluation below a threshold score).

In some implementations, whether or not weaknesses were identified(406), it is determined whether additional skills are being consideredat this time (410). The additional skills, for example, may be relatedskills within a multi-dimensional skill hierarchy. In another example,the additional skills may be other skills of focus to the learner at thepresent time. If there are additional skills, the method 400, in someembodiments, repeats operations 404 through 408 for each additionalskill.

Once all skills have been evaluated (410), in some embodiments, if newcontent was identified in operation 408 (412), at least a portion of theidentified new content is organized for presentation to the learner viathe e-learning platform (418). Identified new content may be rankedand/or sorted, in some examples, by anticipated impact to the learner(e.g., based on strengths and/or weights of the skill in relation to thecontent), by newness to the learner (e.g., as opposed to similar typesof learning resources/resource elements already engaged by the learner),and/or recency of review by the learner (e.g., if the learner hasalready engaged with all content related to the particular skill) Thestudent interactions data 190, for example, may be evaluated for recencyof interactions, types of content the learner has engaged with in thepast, and other metrics applicable to determining best content toprovide to the user to increase mastery of the skills.

If, instead, no new content was identified (412), because the learnerhas already mastered the area of the subject skill(s), in someimplementations, other historic mastery assessments for the learner areaccessed (414) and new content is identified based on weakness(es) inother skill areas (416). The analysis, for example, may be similar tothe analysis described in relation to step 408. The other skill areas,in some examples, may represent next skill areas within a learningstandard, one or more skills tertiary to or building upon the subjectskill(s), or skills related to the subject skill(s) within amulti-dimensional skill hierarchy. The new content, similarly, can beorganized for presentation to the learner (418).

Although illustrated as a particular series of operations, in someembodiments, the operations of the method 400 are performed in adifferent order and/or one or more of the operations of the method 300are performed in parallel. For example, the operations 406 and 408 maybe performed in parallel for each skill of the one or more skills. Insome embodiments, the method 400 includes more or fewer operations.Other modifications of the method 300 are possible while remaining inthe scope and spirit of the disclosure.

In some implementations, the assessment GUI engine 108 b is configuredto present mastery assessments representing data related to a populationof learners. Such assessments may be presented to instructors 154 and/orlearning supervisors 156. In some examples, a particular classroom,grade of a particular school, grade of a particular school district,group of preschoolers at an early learning facility, or other populationmay be tracked for progress over time.

Turning to FIG. 5B, a screen shot of an example skill proficiency flowgraph 520 illustrates mastery assessments measured, for example, on ascale 522 from 0% mastery to 100% mastery and the mastery assessments ofthe learner population are grouped into quartiles of mastery 524 a-d.The quartiles of mastery 154, as illustrated, are separated into a “farbelow skill proficiency” level 524 a, a “below skill proficiency” level524 c, an “approaching skill proficiency” level 524 b, and an “at skillproficiency” level 524 a. Assessment points 526, on the x-axis, areidentified as “after 3 engagement events 526 a, “after 5 engagementevents” 526 b, and “after 10 engagement events” 526 c. An engagementevent, for example, may be qualified as interaction with a learningresource 170/resource element 172 resulting in a skills evaluation 186.

As illustrated in the graph 520, changes in mastery of the skill aretraced for each learner of the population, where skill proficiencies aredemonstrated as improving, remaining substantially the same, and/ordiminishing among members of the population between each pair of theassessment points (points 526 a and 526 b, points 526 b and 526 c). Aslearners interact with additional learning resources 170/resourceelements 172, for example, a difficulty level may increase, for exampleleading to a reduction in perceived skill proficiency despite theadditional exposure to the skill in general. However, overall, for eachlater assessment point 526 b and 526 c, the relative percentage oflearners in the “far below skill proficiency” level 524 d diminishes,and the relative percentage of learners in the “at skill proficiency”level 524 a increases.

In some implementations, the graph 520 is an interactive graph, allowinga reviewer to drill down or obtain additional information regarding themovements of the students. For example, the review of the graph 520 maybe enabled to review numbers of students maintaining level (e.g., 524a-d), increasing level, or decreasing level through hovering over pointson the graph 520.

Reference has been made to illustrations representing methods andsystems according to implementations of this disclosure. Aspects thereofmay be implemented by computer program instructions. These computerprogram instructions may be provided to a processor of a general-purposecomputer, special purpose computer, or other programmable dataprocessing apparatus and/or distributed processing systems havingprocessing circuitry, such that the instructions, which execute via theprocessor of the computer or other programmable data processingapparatus, create means for implementing the functions/operationsspecified in the illustrations.

One or more processors can be utilized to implement various functionsand/or algorithms described herein. Additionally, any functions and/oralgorithms described herein can be performed upon one or more virtualprocessors. The virtual processors, for example, may be part of one ormore physical computing systems such as a computer farm or a clouddrive.

Aspects of the present disclosure may be implemented by software logic,including machine readable instructions or commands for execution viaprocessing circuitry. The software logic may also be referred to, insome examples, as machine readable code, software code, or programminginstructions. The software logic, in certain embodiments, may be codedin runtime-executable commands and/or compiled as a machine-executableprogram or file. The software logic may be programmed in and/or compiledinto a variety of coding languages or formats.

Aspects of the present disclosure may be implemented by hardware logic(where hardware logic naturally also includes any necessary signalwiring, memory elements and such), with such hardware logic able tooperate without active software involvement beyond initial systemconfiguration and any subsequent system reconfigurations (e.g., fordifferent object schema dimensions). The hardware logic may besynthesized on a reprogrammable computing chip such as a fieldprogrammable gate array (FPGA) or other reconfigurable logic device. Inaddition, the hardware logic may be hard coded onto a custom microchip,such as an application-specific integrated circuit (ASIC). In otherembodiments, software, stored as instructions to a non-transitorycomputer-readable medium such as a memory device, on-chip integratedmemory unit, or other non-transitory computer-readable storage, may beused to perform at least portions of the herein described functionality.

Various aspects of the embodiments disclosed herein are performed on oneor more computing devices, such as a laptop computer, tablet computer,mobile phone or other handheld computing device, or one or more servers.Such computing devices include processing circuitry embodied in one ormore processors or logic chips, such as a central processing unit (CPU),graphics processing unit (GPU), field programmable gate array (FPGA),application-specific integrated circuit (ASIC), or programmable logicdevice (PLD). Further, the processing circuitry may be implemented asmultiple processors cooperatively working in concert (e.g., in parallel)to perform the instructions of the inventive processes described above.

The process data and instructions used to perform various methods andalgorithms derived herein may be stored in non-transitory (i.e.,non-volatile) computer-readable medium or memory. The claimedadvancements are not limited by the form of the computer-readable mediaon which the instructions of the inventive processes are stored. Forexample, the instructions may be stored on CDs, DVDs, in FLASH memory,RAM, ROM, PROM, EPROM, EEPROM, hard disk or any other informationprocessing device with which the computing device communicates, such asa server or computer. The processing circuitry and stored instructionsmay enable the computing device to perform, in some examples, the method200 of FIG. 2A, the method 300 of FIG. 3A, the process 320 of FIG. 3B,and/or the method 400 of FIG. 4 .

These computer program instructions can direct a computing device orother programmable data processing apparatus to function in a particularmanner, such that the instructions stored in the computer-readablemedium produce an article of manufacture including instruction meanswhich implement the function/operation specified in the illustratedprocess flows.

Embodiments of the present description rely on network communications.As can be appreciated, the network can be a public network, such as theInternet, or a private network such as a local area network (LAN) orwide area network (WAN) network, or any combination thereof and can alsoinclude PSTN or ISDN sub-networks. The network can also be wired, suchas an Ethernet network, and/or can be wireless such as a cellularnetwork including EDGE, 3G, 4G, and 5G wireless cellular systems. Thewireless network can also include Wi-Fi®, Bluetooth®, Zigbee®, oranother wireless form of communication. The network, for example, maysupport communications between the electronic learning platform 102 andthe computing devices 106 a as shown in FIG. 1A and/or between thestudents 152, instructors 154, and/or learning supervisors 156 and thee-learning platform 102 as shown in FIG. 1B.

The computing device, in some embodiments, further includes a displaycontroller for interfacing with a display, such as a built-in display orLCD monitor. A general purpose I/O interface of the computing device mayinterface with a keyboard, a hand-manipulated movement tracked I/Odevice (e.g., mouse, virtual reality glove, trackball, joystick, etc.),and/or touch screen panel or touch pad on or separate from the display.The display controller and display may enable presentation of the screenshots illustrated, in some examples, in FIG. 5A and FIG. 5B.

Moreover, the present disclosure is not limited to the specific circuitelements described herein, nor is the present disclosure limited to thespecific sizing and classification of these elements. For example, theskilled artisan will appreciate that the circuitry described herein maybe adapted based on changes in battery sizing and chemistry or based onthe requirements of the intended back-up load to be powered.

The functions and features described herein may also be executed byvarious distributed components of a system. For example, one or moreprocessors may execute these system functions, where the processors aredistributed across multiple components communicating in a network. Thedistributed components may include one or more client and servermachines, which may share processing, in addition to various humaninterface and communication devices (e.g., display monitors, smartphones, tablets, personal digital assistants (PDAs)). The network may bea private network, such as a LAN or WAN, or may be a public network,such as the Internet. Input to the system, in some examples, may bereceived via direct user input and/or received remotely either inreal-time or as a batch process.

Although provided for context, in other implementations, methods andlogic flows described herein may be performed on modules or hardware notidentical to those described. Accordingly, other implementations arewithin the scope that may be claimed.

In some implementations, a cloud computing environment, such as GoogleCloud Platform™ or Amazon™ Web Services (AWS™), may be used perform atleast portions of methods or algorithms detailed above. The processesassociated with the methods described herein can be executed on acomputation processor of a data center. The data center, for example,can also include an application processor that can be used as theinterface with the systems described herein to receive data and outputcorresponding information. The cloud computing environment may alsoinclude one or more databases or other data storage, such as cloudstorage and a query database. In some implementations, the cloud storagedatabase, such as the Google™ Cloud Storage or Amazon™ Elastic FileSystem (EFS™), may store processed and unprocessed data supplied bysystems described herein. For example, the contents of the data store104 of FIG. 1A, the contents of the e-resource data store 158 and/or theuser data store 160 of FIG. 1B, and/or the contents of the historicinteraction data 322, the skills hierarchy 174, the tagged learningresource elements 324, and/or the mastery assessment parameters 180 ofFIG. 3B may be maintained in a database structure.

The systems described herein may communicate with the cloud computingenvironment through a secure gateway. In some implementations, thesecure gateway includes a database querying interface, such as theGoogle BigQuery™ platform or Amazon RDS™. The data querying interface,for example, may support access by the e-learning platform 102 to thee-resource data 158 and/or the user data 160. In another example, thedata querying interface may support access by the evaluation predictionengine 162 to the historic interaction data 322, tagged learningresource elements 324, skills hierarchy 174, student interactions data190, and/or mastery assessment parameters 180, as shown in FIG. 3B.

While certain embodiments have been described, these embodiments havebeen presented by way of example only and are not intended to limit thescope of the present disclosures. Indeed, the novel methods, apparatusesand systems described herein can be embodied in a variety of otherforms; furthermore, various omissions, substitutions and changes in theform of the methods, apparatuses and systems described herein can bemade without departing from the spirit of the present disclosures. Theaccompanying claims and their equivalents are intended to cover suchforms or modifications as would fall within the scope and spirit of thepresent disclosures.

1. (canceled)
 2. A system for progressing learners through a pluralityof electronic learning resources of an e-learning platform to effectmastery of one or more skills of a skill set, the system comprising: atleast one non-volatile computer readable medium configured to store theplurality of electronic learning resources, each respective learningresource having at least one logical skill-item connection of aplurality of skill-item connections, each logical skill-item connectionof the at least one logical skill-item connection connecting therespective learning resource to a corresponding skill of a plurality ofskills, wherein the plurality of skills are logically arranged in askills hierarchy comprising a plurality of skill families, each skillfamily comprising logical connections forming descendant and ancestorrelationships between a respective set of skills of the plurality ofskills belonging to the respective skill family, each logical skill-itemconnection of at least a portion of the plurality of skill-itemconnections is associated with a skill-item weight of a plurality ofskill-item weights, the skill-item weight representing a relative impactof skill development in the corresponding skill imparted throughinteractions with the respective electronic learning resource, and atleast a portion of the plurality of electronic learning resources areeach logically connected to two or more skills of the plurality ofskills, wherein each set of logical skill-item connections of theplurality of skill-item connections connecting a respective two or moreskills of the plurality of skills to a same electronic learning resourceof the plurality of electronic learning resources is associated with atleast one skill-item strength of a plurality of skill-item strengthsrepresenting a relative relevance of each skill of the respective two ormore skills; and processing circuitry configured to perform operationscomprising automatically evaluating interactions produced by a givenlearner of a plurality of learners engaging with a given learningresource of the plurality of learning resources, wherein the evaluatingcomprises applying evaluation rules for the given learning resource tothe interactions, and the evaluating results in a skill assessment of aplurality of skill assessments of the given learner, determining alearner group of a plurality of learner groups applicable to the givenlearner, accessing, based on the learner group, a set of masteryassessment parameters, wherein the set of mastery assessment parametersis one of a plurality of sets of mastery assessment parameters, each setof the plurality of sets of mastery assessment parameters correspondingto a different learner group of the plurality of learner groups, whereinthe set of mastery assessment parameters comprises at least a firstmastery assessment parameter related to the plurality of skill-itemweights and at least a second master assessment parameter related to theplurality of skill-item strengths, using the set of mastery assessmentparameters, automatically analyzing a set of skill assessments of theplurality of skill assessments of the given learner corresponding to agiven skill family of the plurality of skill families to produce amastery assessment corresponding to a skill area of the given skillfamily, and selecting, based at least in part on the mastery assessment,the one or more electronic learning resources of the plurality ofelectronic learning resources for strengthening the mastery of the skillarea.
 3. The system of claim 2, wherein the plurality of learner groupsis divided by one or more of age ranges, grade levels, geographicregions, or learning standards.
 4. The system of claim 2, whereinapplying the evaluation rules comprises mathematically characterizingportions of the interactions as at least one of desired, undesired,correct, or incorrect.
 5. The system of claim 2, wherein: at least aportion of the interactions each correspond to a respective timestamp ofa plurality of timestamps; and applying the evaluation rules comprisescalculating at least one elapsed time using the plurality of timestamps,and characterizing the interactions in part based on the elapsed time.6. The system of claim 2, wherein applying the evaluation rulescomprises producing at least one count of a portion of the interactionscorresponding to a series of user inputs.
 7. The system of claim 2,wherein the set of skill assessments corresponds to a sub-hierarchy ofthe skill family representing at least one of a skill type or a skillsubject.
 8. The system of claim 2, wherein the operations furthercomprise calculating a confidence value representing a relativeconfidence in the mastery assessment accurately capturing a presentlevel of the mastery of the given learner in the skill area.
 9. Thesystem of claim 2, wherein automatically analyzing the set of skillassessments of the given learner comprises analyzing the set of skillassessments in view of a time progression of the set of skillassessments.
 10. The system of claim 2, wherein selecting the one ormore electronic learning resources comprises identifying the one or moreelectronic learning resources as having at least one of a) a respectiveskill-item weight at or above a first threshold value or b) a respectiveskill-item strength at or above a second threshold value.
 11. The systemof claim 2, wherein the operations further comprise organizing the oneor more electronic learning resources for presentation to the givenlearner via the e-learning platform.
 12. A method for progressinglearners through a plurality of electronic learning resources of ane-learning platform to effect mastery of one or more skills of a skillset, the method comprising: storing, to non-volatile computer-readablemedia of the e-learning platform, the plurality of electronic learningresources, each respective learning resource having at least one logicalskill-item connection of a plurality of skill-item connections, eachlogical skill-item connection of the at least one logical skill-itemconnection connecting the respective learning resource to acorresponding skill of a plurality of skills, wherein the plurality ofskills are logically arranged in a skills hierarchy comprising aplurality of skill families, each skill family comprising logicalconnections forming descendant and ancestor relationships between arespective set of skills of the plurality of skills belonging to therespective skill family, each logical skill-item connection of at leasta portion of the plurality of skill-item connections is associated witha skill-item weight of a plurality of skill-item weights, the skill-itemweight representing a relative impact of skill development in thecorresponding skill imparted through interactions with the respectiveelectronic learning resource, and at least a portion of the plurality ofelectronic learning resources are each logically connected to two ormore skills of the plurality of skills, wherein each set of logicalskill-item connections of the plurality of skill-item connectionsconnecting a respective two or more skills of the plurality of skills toa same electronic learning resource of the plurality of electroniclearning resources is associated with at least one skill-item strengthof a plurality of skill-item strengths representing a relative relevanceof each skill of the respective two or more skills; automaticallyevaluating, by processing circuitry, interactions produced by a givenlearner of a plurality of learners engaging with a given learningresource of the plurality of learning resources, wherein the evaluatingcomprises applying evaluation rules for the given learning resource tothe interactions, wherein the evaluating results in a skill assessmentof a plurality of skill assessments of the given learner, determining,by the processing circuitry, a learner group of a plurality of learnergroups applicable to the given learner, accessing, by the processingcircuitry based on the learner group, a set of mastery assessmentparameters, wherein the set of mastery assessment parameters is one of aplurality of sets of mastery assessment parameters, each set of theplurality of sets of mastery assessment parameters corresponding to adifferent learner group of the plurality of learner groups, wherein theset of mastery assessment parameters comprises at least a first masteryassessment parameter related to the plurality of skill-item weights andat least a second master assessment parameter related to the pluralityof skill-item strengths, using the set of mastery assessment parameters,automatically analyzing, by the processing circuitry, a set of skillassessments of the plurality of skill assessments of the given learnercorresponding to a given skill family of the plurality of skill familiesto produce a mastery assessment corresponding to a skill area of thegiven skill family, and providing, by the processing circuitry, accessvia the e-learning platform to a graphical analysis of the masteryassessment for review by an instructor or a learning supervisor of thegiven learner.
 13. The method of claim 12, wherein the plurality oflearner groups is divided by one or more of age ranges, grade levels,geographic regions, or learning standards.
 14. The method of claim 12,wherein the evaluation rules are configured to adjust the skillassessment based in part on partial completion of interacting with thegiven learning resource.
 15. The method of claim 12, wherein: the givenelectronic resource is logically connected to at least two skills; andthe evaluating results in the skill assessment corresponding to a firstskill of the at least two skills and a second skill assessmentcorresponding to a second skill of the at least two skills.
 16. Themethod of claim 15, wherein applying the evaluation rules comprisesapplying a first portion of the evaluation rules to a first subset ofthe interactions corresponding to the first skill and applying a secondportion of the evaluation rules to a second subset of the interactionscorresponding to the second skill.
 17. The method of claim 12, furthercomprising: using a plurality of sets of historic time series ofinteractions with a portion of the plurality of electronic learningresources interactions related to a given skill family of the pluralityof skill families, each set belonging to a separate learner of theplurality of learners, training, by the processing circuitry, at leastone machine learning algorithm to forecast mastery assessment outcomesof learning resource; and applying the at least one machine learningalgorithm to a plurality of time series of interactions a with one ormore electronic learning resources of the plurality of electroniclearning resources related to the given skill family, each set belongingto a different learner of a set of learners, to produce predictedevaluation outcomes corresponding to the plurality of time series ofinteractions.
 18. The method of claim 17, further comprising: comparing,by the processing circuitry, the predicted evaluation outcomes withactual evaluation outcomes for the set of learners; and based on one ormore differences between the predicted evaluation outcomes and theactual evaluation outcomes, refining, by the processing circuitry, atleast one parameter of the set of mastery assessment parameters.
 19. Themethod of claim 17, wherein: a first machine learning algorithm of theat least one machine learning algorithm is a regression-based model; andthe first machine learning algorithm is trained to forecast theevaluation outcomes based at least in part on engagement patterns andengagement timings indicative of one or more of skill fading, impartinglearning, initial level of mastery, or a difficulty of acquiringmastery.
 20. The method of claim 12, wherein the plurality of electroniclearning resources comprises a plurality of questions, a plurality ofvideos, and a plurality of games.
 21. The method of claim 12, whereinthe interactions comprise at least one of text engagements, verbalengagements, or gesture engagements.